Free Energy in Statistical Physics and Inference

Random notes on the useage of the free energy in statistical physics and inference. Taken from David's book Chapter 31 on Ising models, Radford Neal's review of MCMC methods, and Yedidia, Freeman and Weiss's introduction to understanding belief propogation and its generalisations. First of all we briefly summarise statistical physics: micro-state = exact microscopic description of a physical system (precise velocity and position of each molecule in a gas for example) macro-state = description of a physical system sufficient to determine any macroscopic observable (the temperature, volume and mass of the gas, for example) The micro-state is generally unknowable and therefore must be handled using probabilites. One of the goals of statistical physics is to related these two levels of description Every possible microstate s of the system has some definite energy E(s). If the system is isolated then the energy is fixed at E 0 , and the assumption is generally made that all microstates with that energy are equally likely: P (s) = Z −1 δ[E 0 − E(s)] (which is the microcanonical distribution). If the system can exchange energy with a large reservoir that maintains the system at constant temperature, the system's energy can fluctuate and it is assumed: P (s) = Z −1 exp[−βE(s)], where the nor-malising constant is Z = ˜ s exp[−βE(˜ s)]. This is called the canonical, Boltzmann or Gibbs distribution over microstates. Intensive quantities are independent of system size (eg. temperature) Extensive quantities grow with system size eg. energy. If interactions are local then the growth will be linear for large systems and the values of extensive quantity per unit of system will reach limiting values. Extensive quantities are interesting at phase transitions. Theory of universality: All systems with the same dimension and same symmetries have equivalent critical properties (ie. the scaling laws shown by their phase transitions are identical)